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Abstract 

Translation and rotation numbers have played an interesting and im- 
portant role in the qualitative description of various dynamical systems. 
In this exposition we are especially interested in applications which lead 
to proofs of periodic motions in various kinds of dynamics on the annulus. 
The applications include billiards and geodesic flows. 

Going beyond this simple qualitative invariant in the study of the dy- 
namics of area preserving annulus maps, G.D. Birkhoff was led to the 
concept of "regions of instability" for twist maps. We discuss the closely 
related notion of instability sets for a generic area preserving surface dif- 
feomorphism and develop their properties. 

1 Introduction 

The concepts of translation number and rotation number were introduced by 
Henri Poincare and have had a rich history. We describe several of many mani- 
festations of these concepts as they apply to the qualitative description of various 
dynamical systems. We are especially interested in applications which lead to 
proofs of periodic motions in various kinds of area preserving dynamics on the 
annulus and S 2 . Applications include billiards and geodesic flows. This is the 
content of sections §2 to §5. 

In pursuit of the qualitative analysis of such dynamics, G.D. Birkhoff was 
led to the concept of "regions of instability" for twist maps on an annulus. We 
discuss the closely related notion of instability sets for a generic area preserving 
surface diffeomorphism and describe some of their properties. 

Guiding this is the deep theory of twist maps of the annulus where we have 
a large body of knowledge, thanks to the pioneering work of G.D. Birkhoff 
and subsequent work by Arnold, Kolmagorov, Herman, Mather, and Moser. 
However, the definition of an annular twist map depends on a distinguished 
choice of co-ordinates on the annulus. In a local setting, e.g. in the neighborhood 
of a generic fixed point, one can always obtain these co-ordinates, but they are 
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generally unavailable in the larger global context. It would be nice to have the 
same or analogous results in this global context. 

In sections § El § 13 and § [S] we describe a residual class of C r area preserv- 
ing surface diffeomorphisms for r > 16 and describe some of their qualitative 
properties. 

In §9 we discuss some of the intriguing open questions related to instability 
sets. 

2 Circle Homeomorphisms 

The simplest instance of rotation number is its use as an invariant for orientation 
preserving circle homeomorphisms. Define the circle T 1 to be the quotient, 
R/Z. As a quotient T 1 inherits both a group structure and a topology. There 
is a natural projection tt : K — > T 1 defined by tt(x) = x + Z which is both a 
(universal) covering map and a group homomorphism. 

Definition 2.1. Let f : T 1 —> T 1 be an orientation preserving homeomorphism 
of the circle. A homeomorphism F : R — > R is a lift of f provided 

f(n(t))=n(F(t)). 

If F is a lift of / it is not difficult to see that G : R — > R is also a lift of / if 
and only if F(t) — G(t) + n for some neZ. 

Definition 2.2. Let f : T 1 — ► T 1 be an orientation preserving homeomorphism 
and let F : R — > R be a lift. The translation number 1 of a point iel under F 
is 

( ^ r Fn ( x )-* 
T{x,b)= nm . 

n — >oo tt, 

Notice that the value of r(x, -F) is the average amount the point x is trans- 
lated under iteration by F . We summarize the elementary properties of the 
translation number of a lift F in the following proposition (See section 11.1 of 
14 for a proof). 

Proposition 2.3. Let f : T 1 — > T 1 6e an orientation preserving homeomor- 
phism and let F : R — > R &e a Z«/f. 

• T/ie Zimit r(x, -F) exists for all x £ R. 

• t(:b, F") = nr(x, F) for all n E Z. 

• t(x, F + m) = t{x, F) + m /or m G Z. 

• t(F) := r(a;,F) is independent of x. 

x It is common to see this quantity referred to as the rotation number, however, the term 
translation number, while less commonly used, is more accurately descriptive. More impor- 
tantly, there is a need to distinguish the translation number from its projection in T 1 , which 
is properly called the rotation number (see Definition (12.51 ~1 . 
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More striking is the fact that if t(F) is rational, then there are of periodic 
orbits. 

Theorem 2.4. Let f : T 1 — > T 1 be an orientation preserving homeomorphism 
and let F :R^R be a lift. 

• t(F) — if and only if F has a fixed point. 

• f has a fixed point if and only if t(F) G Z. 

• / has a periodic point of period q if and only if t(F) = p/q G Q, where 
p/q is in lowest terms. 

If F and G are two different lifts of / : T 1 — > T 1 , then they differ by a 
constant integer, i.e. there is an m G Z such that F(t) = G(t) + m for all t£R. 
From this and the proposition above it follows that t(x, F) = t(x, G) + m. As 
a consequence if we reduce t(x,F) modulo 1, or, more precisely, consider its 
projection in T 1 := R/Z the result is independent of the choice of the lift F and 
depends only on the original homcomorphisms /. 

Definition 2.5. Let f : T 1 — > T 1 be an orientation preserving homeomorphism 
and let F : R — » R be a lift. The rotation number of f is the element of T 1 
defined by 

p(J)=*(T(x,F)) 

where ir : K — > T 1 is f/ie natural projection, F is any lift of f and x is any point 
ofR. 

From Proposition 12.31 we immediately obtain the following 

Proposition 2.6. Let f : T 1 — > T 1 be an orientation preserving homeomor- 
phism. 

• p{f n ) = np(F) for all n G Z. 

• / has a periodic point of period q if and only if p(f) = (p/q + Z) G Q/Z 
where p/q is in lowest terms. 

For the many important dynamical systems which preserve a measure there 
is often an important relation between the measure and the rotation numbers. 
In the case of circle homomorphisms this relation has an especially simple form 
which we will now describe. 

Definition 2.7. Let f : T 1 — > T 1 be an orientation preserving homeomorphism 
and let F : R — > R be a lift. We define the displacement function (j> : T 1 — » R of 
F by (f>{x + Z) = F{x)-x. 

Note that if x' is another choice for x, (i.e. x + Z = x' + Z), then a;' = a; + m 
for some integer to so i^(a;) — x = F(x') — x' and <^ is well defined. For a point 
(x + Z) G T 1 the value of <f> can be thought of as amount this point is displaced 
around the circle. Having chosen a lift F we have determined this as a well 
defined element of R, not just an element of R modulo 1. 
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Theorem 2.8. Let f : T 1 — ► T 1 be an orientation preserving homeomorphism 
which preserves a probability measure fi, and let F be a lift of f and suppose 
4> : T 1 — > M is the displacement function of F. Then 

r(F) = [ 4>dpi. 

Notice that the left hand side of this equality is independent of /x and hence 
the right hand side is the same for all choices of fj,\ 

Proof. Here we use z to denote a point of T 1 instead of the more cumbersome 
x + Z. According to the Birkhoff Ergodic Theorem (see the function <j> 

defined by 

n— 1 

${z)= lim -J2<P(f(z)) 

n — >oo fi — 4 

i=0 

is integrable and satisfies 




But if z = n(x), then f l (z) = -k{F 1 (z)) and 4>{P{z)) = F(F z (x)) - F z (x) = 
F l+1 (x) — F t (x). It follows that the telescoping sum 

n— 1 n— 1 

£ Hf(z)) = ^ {F i+1 (x) - F\x)) = F n (x) x. 

i=Q i=0 

Hence 

0( z ) = lim Fn{ - x) ~ X = t(F). 

n— >oo /j 

So is the constant function with value t(F) and the result follows, o 



3 Annulus Homeomorphisms 

The annulus A = T 1 x I, where I — [0, 1], provides much richer ground for 
application of the concept of rotation number. 

Before exploring this, however, we motivate our interest in the annulus by de- 
scribing one of the many mechanical systems which give rise to annulus maps. 
Poincare's original interest in rotation numbers was related to the restricted 
three body problem, but, a system much easier to describe is provided by bil- 
liards on a strictly convex table with smooth boundary. 

The ball is idealized as a point which moves on the table with uniform 
motion and bounces off the edge in such a way that the angle of incidence equals 
the angle of reflection. The annulus associated with this dynamical system is 
T 1 x [0, 7r] where the circle T 1 parametrizes the edge of the table and 9 € [0, ir] is 
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the angle between the tangent to the table edge and the direction at which the 
the ball leaves the edge at a bounce. If a; is a point of contact and 6 € (0, tt) is 
an angle of reflection, then the billiard map F : A — > A is defined by F(x, 9) = 
(x',9'), where x' is the next point of contact with the boundary and 9' is the 
next angle of reflection. Notice that by defining F(x, 0) = x and F(x,ir) = x 
we can extend F to a homeomorphism of all of A. 

To understand the billiard dynamics qualitatively (for example the existence 
of periodic orbits) it suffices to understand this billiard map. A crucial aid in 
this analysis comes from the fact the map preserves an area. 

Theorem 3.1 (Birkhoff). If we define an area on A using the element of area 
sin (9 dx d9, then this area is preserved by the billiard map F : A — > A. 

As in the case of a circle we can define the notions of translation number 
and rotation number. 

Definition 3.2. Let A = M x I. A homeomorphism F 
provided 

f(n(x,y)) = n(F(x,y)), 
where n : A — > A is given by tt(x, y) = (vr(a;), y). 

If F is a lift of / it is not difficult to see that G : A - 
and only if 

F(x,y) = G(x,y) + (m,0) 

for some raeZ. 

Definition 3.3. Let f : A — » A be an orientation preserving homeomorphism 
which preserves boundary components and let 

F : A — > A be a lift. The translation number of a point w — (x.y) £ A under F 
is 

Pl {F n {w)-w) 
T[w,h)= hm , 

where p\ : A — > K is projection on the first co-ordinate. 

We would like an analogue of Proposition 12.31 i.e. we would like for the 
properties of the translation number for circle homeomorphisms to hold for 
annulus homeomorphisms. Unfortunately, much less is true for the annulus. In 
particular t(w, F) may fail to exist for many w G A. Moreover, when it does 
exist, it is only rarely independent of w. 

Proposition 3.4. Let f : A — > A be an orientation preserving homeomorphism 
which preserves boundary components and let F : A — > A be a lift. 

• If t(w, F) exists, then t{w, F n ) = nr{w, F) for all n G Z. 

• // t(w, F) exists, then t{w, F + (m, 0)) = t(w, F) + m for m G Z. 

• If f preserves a finite Borel measure /i and fi is the lift of this measure to 
A, then t(w, F) exists for all w except a set of fi measure zero. 



: A — > A is a lift of f 



A is also a lift of / if 
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The proofs of the first two parts of this proposition are straightforward. The 
final part is a consequence of the Birkhoff crgodic theorem and we give a proof 
below in Theorem 13.81 

Definition 3.5. Let f : A — » A be an orientation preserving homeomorphism 
and let F : A — > A be a lift. The rotation number of f is the element of T 1 
defined by 

p(f)=ir( T (x,F)) 

where it : R — > T 1 is the natural projection, F is any lift of f and x is any point 
ofR. 

From Proposition 13.41 we immediately obtain the following: 

Proposition 3.6. Let f : A — ► A be an orientation preserving homeomorphism 
which preserves boundary components and let F : A — > A be a lift. 

• Ifr(w,F) exists, then p(w(w),f) exists and is well defined, i.e. indepen- 
dent of the lift F. 

• If p(x, f) exists p(w, /") = np(x, f) for all n£Z. 

• If f preserves a finite Borel measure p then p{x, f) exists for all x G A 
except a set of p measure zero. 

It is not hard to see that periodic orbits have rational translation numbers. 
But the issue of using translation numbers or rotation numbers to prove the 
existence of periodic points is much more subtle than in the case of the circle 
and that is the subject of 2| below. 

The last item in Proposition l3 . 61 makes it very important to have a measure 
preserving annulus homeomorphism and it is in this setting that we are able to 
provide the most interesting results. 

As in the case of T 1 it is useful to define the displacement function for a lift 

F. 

Definition 3.7. Let f : A — » A be an orientation preserving homeomorphism, 
preserving boundary components, and let F : A — » A be a lift. We define the 
displacement function <f> : A — > K of F by <P(tt(w)) = pi(F(w) — w) for w 6 A, 
where p\ : A — > R is projection on the first co-ordinate. 

Note that if w' satisfies tt(w) = tt(w') then w' = w + (to, 0) for some integer 
m so F(w) — w = F(w') — w' and <fi is well defined. 

Theorem 3.8. Let f : A — > A be an orientation preserving homeomorphism, 
preserving boundary components, and preserving a probability measure p, and 
let F be a lift of f . Then the rotation number p(x,F) exists for p-almost all x 
and is a p integrable function. If (f> : A — > R is the displacement function of F , 

I p(x, F) dp = 4> dp. 
Jt 1 Jt 1 

The value of these integrals will be denoted by t^{F) and is called the mean 
translation number of F . 
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Proof. Again we appeal to the Birkhoff Ergodic Theorem (see 0]). As in the 
proof of Theorem 12. 81 if x = tt(w), then the telescoping sum 

n—l n—1 

£ M 1 ^)) = E^ +1 W " F * W) - Pi(F"(w) - w). 

i=0 i=0 

So 

t{w,f) = to ftgMz^ a i im 

Hence, according to the ergodic theorem, r(u>, i* 1 ) exists for /i-almost all a; and 
is a integrable. Moreover it satisfies 

Tp(F) = t dn= 4> dfi. 
Jt 1 Jt 1 

o 



4 The Poincare-Birkhoff Theorem 

In this section we describe several theorems analogous to Theorem 12.41 in the 
sense that from a hypothesis on a translation number one is able to conclude 
the existence of a periodic orbit. The most famous of these is the Poincare- 
Birkhoff Theorem. This was conjectured by Poincare and he was able to prove 
some special cases. His motivation was to prove the existence of some periodic 
motions of the restricted three-body problem (this is the Newtonian motion of 
three point masses, one of which has zero mass). 

G. D. Birkhoff who succeeded in proving the result Poincare conjectured,, in 
1 . This was perhaps the first of Birkhoff 's many great contributions to bring 
him world recognition in the mathematical community. 

Theorem 4.1 (G. D. Birkhoff). // / : A — > A is an area preserving annulus 
homeomorphism which preserves orientation and boundary components. Let 
F : A ^ A be a lift and let F (x) = F(x,0), F 1 (x) = F(x, 1). Ifp/q e Q and 

t(F ) <^< T (Fi) 

then F has at least two points with translation number p/q which are lifts of 
periodic points of f with distinct orbits of period q and rotation number p/q + Z. 

In addition to the application which Poincare had in mind this result can 
be applied to billiards on a table with smooth convex boundary as described 
above. In fact, if / : A — > A is the billiard map, there is a lift F : A — > A with 
the property that for all x G M, Fq(x) = x, and F±(x) = x + 1. It follows that 
T (Fo) = and t(F\) — 1 so according to the theorem for every p/q £ [0, 1] there 
are two periodic orbits with translation number p/q and rotation number p/q+Z. 
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This is true for any billiard table as long as it has a smooth strictly convex 
boundary. (In fact, strict convexity is not really necessary; simple convexity 
and smoothness of the boundary suffices.) 

Figure 1 shows examples with two periodic billiard orbits on a convex table 
with rotation numbers p(x, /) = i + Z and p(x, f) = | + Z. 




Figure 1: Periodic billiard orbits with p(x, /) = | + Z and p(x, /) = | + Z 

The Poincare-Birkhoff theorem has a number of generalizations. One which 
is important for our purposes asserts that we do not need information about 
the behavior of F on the boundary of A, it suffices to have any two points 
wq,Wi € A with distinct translation numbers to obtain periodic lifts with all 
rational translation numbers in the interval [t(wq, F), t(w%, F)]. 

Theorem 4.2 (Franks |3"].pEp. Suppose f : A — > A is an area preserving 
annulus homeomorphism which preserves orientation and boundary components. 
Let F : A — > A be a lift. If wq, W\ G A and p/q £ Q satisfy 

t{w ,F) < V - <t(w u F) 

then F has at least two points with translation number p/q which are lifts of 
periodic points of f with distinct orbits of period q and rotation number p/q + Z. 

Of course, it is natural to ask about points with irrational translation num- 
ber. 

Theorem 4.3 (M. Handel [12 ). Suppose that the homeomorphism f : A — > A 
preserves orientation and boundary components and F is a lift. Then the set 
{t(w, F) w £ A} is closed. 

Notice that unlike the Poincare-Birkhoff Theorem, no area preserving hy- 
pothesis is necessary for Handel's result. However when the area preserving 
hypothesis holds, we can conclude from the results above that the set of all 
translation numbers of a lift F forms a closed interval. 

Finally we note that there is also a result relating rationality of the mean 
translation number to the existence of periodic points. 
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Theorem 4.4 (Franks [5j). Suppose f : A — ► A is an annulus homeomorphism 
which preserves orientation and boundary components. Let F : A — > A be a lift. 
If f preserves Lebesgue measure fi and 

q 

then F has at least two points with translation number p/q which are lifts of 
periodic points of f with distinct orbits of period q and rotation number p/q + Z. 

5 The geodesic return map on S 2 

In this section we discuss another application of translation and rotation num- 
bers, namely the problem of the existence of closed geodesies on the two-sphere 
S 2 . 




Figure 2: The Geodesic Return Map 



One of the standard ways to find closed geodesies is to try to minimize 
the length of a closed curve in its homotopy class. Of course, this cannot 
immediately be applied to S 2 since all closed curves are homotopic to curves 
of arbitrarily small length. Nevertheless a fascinating result of Birkhoff, using 
a mini-max argument, does succeed in obtaining at least one closed geodesic 
for S 2 provided with an arbitrary Riemannian metric. In fact the geodesic so 
obtained is simple, i.e. does not cross itself. 

Theorem 5.1 (Birkhoff). For any Riemannian metric on S 2 there is always 
at least one simple closed geodesic. 
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In fact, Liusternik and Schnirrclman subsequently proved that there are at 
least three simple closed geodesies on such a sphere. 

Birkhoff realized that, like the problem of periodic billiards, the existence of 
closed geodesies could often be reduced to the study of an annulus map. 

Definition 5.2. The geodesic return map 

F : A — > A for a simple closed geodesic g C S 2 is defined by 

F(x,e) = ( x ',e'), 

where x' is the second point where g is crossed by a geodesic which starts at x 
and forms angle 9 with g. 9' is the new angle. (See Figure 2). 

Theorem 5.3 (Birkhoff). The geodesic return map is well-defined for a sphere 
if it has positive curvature. Whenever it is well defined it is an area preserving 
diffeomorphism. 

This motivates an interest in the existence of periodic orbits for area pre- 
serving annulus maps since there is a one-to-one correspondence between closed 
geodesies and periodic orbits of the geodesic return map. The following result 
shows, in particular, that whenever the geodesic return map is well-defined for 
a Riemannian metric on S 2 , there are infinitely many distinct closed geodesies 
for that metric. 

Theorem 5.4 (Franks [5 ). Suppose f : A — > A is an area preserving annulus 
diffeomorphism which preserves orientation and boundary components. If f has 
at least one periodic point, then it has infinitely many. 

Theorem 5.5 (V. Bangert (J). When the geodesic return map is not well 
defined there are infinitely many closed geodesies on S 2 . 

Combining these results we have the following result which answers a long- 
standing question about the existence of closed geodesies on S 2 . 

Theorem 5.6. Every smooth Riemannian metric on S 2 has infinitely many 
closed geodesies. 

Shortly after the appearance of this result a rather different proof using 
Morse theory on the space of free loops on S 2 was provided by Nancy Hingston 

E3- 

Theorem 15.41 can be strengthened in several ways. For example in |Hj the 
following result is proved. 

Theorem 5.7 (Franks and Handel, |9J). Suppose 

f : A -> A 

is an area preserving annulus homeomorphism which preserves orientation and 
boundary components. If f has at least one periodic point, then the periods of 
periodic points are unbounded. 
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In fact this result is valid for more general surfaces. An area preserving 
diffeomorphism of a compact surface is called a Hamiltonian diffeomorphism if 
it is a commutator in the group of area preserving diffeomorphisms homotopic 
to the identity. In 3| it is shown that any Hamiltonian diffeomorphism of a 
surface of genus > 1 must have periodic points of arbitrarily high period. 

6 The Instability Set 

The purpose of this section is to delve more deeply into the investigation of the 
dynamics of typical area preserving diffeomorphisms of annuli and spheres. 
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Figure 3: The Standard Map, k = 0.15 

A rather remarkable qualitative behavior is observed if one does graphical 
computer studies of even simple diffeomorphisms of this type. The standard 
map family f : T 1 xR-tT 1 x 1 is given by 

f(x, y) = (x + y + k sin(27ra;), y + k sin(27ra;)) , 

where A; is a parameter. One checks easily that the determinant of the Jacobian 
for this diffeomorphism of the open cylinder is 1, so it is an area preserving 
diffeomorphism. If k = this map exhibits a rather simple dynamics, namely 
each circle T 1 x y in the cylinder is invariant and the restriction of / to this 
circle is a rotation by y$. 

If the parameter k is increased the dynamics becomes much more com- 
plicated. For example Figure 3 shows a computer generated image of sev- 
eral thousand points on an orbit for the standard map with k = 0.15. Actu- 
ally, f(x,y) is periodic of period one, and what has been drawn is the region 
< x < 2, —0.5 < y < 0.5 in order to give a more informative picture. 

11 



"standard-map" 
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The striking feature of this image is that the plotted points (a finite set) 
appear to be a connected set which is riddled with holes. The remarkable 
property of the image is that if another image were created by plotting points 
on the orbit of another point near this set (i.e. not in one of the holes), then the 
image obtained would be visually essentially identical. This is despite the fact 
that it consists of a disjoint set of points. As we will see, it is likely that there 
is a compact connected invariant set which these plots are approximating. 

If we were to fill all the holes in this figure and take the interior of the 
resulting set we would obtain an example of what Birkhoff called a "region of 
instability". Our aim however, is to investigate the black set in the figure which 
we will define, and refer to as an instability set. 

Of particular importance in our investigation are the stable and unstable sets 
which we now define. 

Definition 6.1. Suppose f : X — > X is a homeomorphisms and P C X is a 
compact f -invariant set. The stable set W S (P) is defined by 

W S (P) :={i£l | lim dist( f n (x),P) = 0}, 

n — >oo 

and the unstable set W U (P) is defined by 

W U {P) :={xeX | lim dist(/ n (ic), P) = O}, 

n — > — oo 

where dist(x,P) is the distance from x to the set P. 

If / : S — + S is an area preserving diffeomorphism of a surface and p G S 
is a fixed point, then p is called elliptic if the eigenvalues of Df p are complex 
conjugates of modulus 1 and are not equal to ±1. The point p is hyperbolic if 
the eigenvalues of Df p are real and have modulus distinct from 1. Since the 
diffeomorphism is area preserving and orientation preserving the product of the 
eigenvalues = det(Df p ) = 1. 

If p is hyperbolic, then its stable set is an injectively immersed copy of R, as 
is its unstable set, and these are usually referred to as the stable and unstable 
manifold of p. If p is a periodic point of period n and a hyperbolic fixed point of 
then W s (p) and W w (p) will denote the stable and unstable manifold with 
respect to f n . 

We will say that an elliptic periodic point p of period n is Moser stable if it 
admits a fundamental system of neighborhoods which are closed disks D such 
that f n \dD is minimal. A consequence of KAM theory is that this is the typical 
behavior for elliptic periodic points (see Theorem 16.31 below) . 

Definition 6.2. Suppose f is an area preserving diffeomorphism of an annular 
domain Ao and Per 9 (/) is the set of periodic points of f with period q. Assume 
that for all q 

i) the set Per g (/) is empty or finite and all periodic points are elliptic or 
hyperbolic. 
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ii) every elliptic periodic point in Per g (/) is Moser stable, and the invariant, 
f minimal, Jordan curves bounding neighborhoods of an elliptic periodic 
point z have rotation numbers which are not constant in any neighborhood 
of z, and 

Hi) the intersection of any branches of hyperbolic periodic points are trans- 
verse and any two branches of the same periodic point have non-empty 
intersection. 

Then f will be called Moser generic. 

Combining the work of many authors one can conclude the following 

Theorem 6.3. If £ = S 2 or A and r > 16, then the set of Moser generic 
diffeomorphisms of is a residual subset of of the group DifF£,(£) of C r area 
preserving diffeomorphisms. 

We remark that the high smoothness appears to be required to have all 
elliptic points be Moser stable and the best published result of which we are 
aware is Douady which requires r > 16. The genericity of the properties of 
hyperbolic periodic points of a Moser generic diffeomorphism follow from the 
Kupka-Smale theorem together with results of Mather [TJ\, Pixton [T^l, an( A 
Robinson (cf. Theorem 18 . 71 below 1 . 

The fact that the dynamics of the typical area preserving diffeomorphism is 
radically different from the dynamics of the typical dissipative diffeomorphism 
is amply illustrated in the following result of Mather. 

Theorem 6.4 (Mather [17 ). Ifp is a hyperbolic fixed point of a Moser generic 
diffeomorphism of S 2 , then 

cl{W s {p)) = cl(W u (p)). 

For a Moser generic diffeomorphism there are homoclinic points (i.e. points 
of W s (p) fl W u (p) in both components of W s (p) \ {p}, but the reader should 
be cautioned that we do not know if the homoclinic points are dense in W s (p). 
In any case, this common closure of the stable and unstable manifold is a kind 
of organizing center for the dynamics. Figure 3 was constructed by plotting an 
orbit of the standard map, but had we picked a fixed point and plotted points on 
its stable manifold we would have obtained a visually indistinguishable picture. 
All this motivates the following definition. 

Definition 6.5. Suppose f : S 2 — > S 2 is an area preserving diffeomorphism 
which is Moser generic and p is a hyperbolic point. The instability set A asso- 
ciated to p is defined by 

A = cl(W s ( P )) =cl(W u ( P )). 

The most fundamental open question concerning Moser generic diffeomor- 
phisms would be answered by a proof of the following conjecture. 
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Conjecture 6.6. If f : S 2 — > S 2 is an area preserving diffeomorphism which is 
Moser generic, then Per(/) is dense in S 2 . 

Of course we would be satisfied with the proof that some other residual 
subset of Diff^(5 2 ) with r arbitrarily large has the property that all of its 
elements have a dense set of periodic points. This conjecture is intimately 
related with the question of whether there is a C r "closing lemma" for area 
preserving C r diffeomorphisms when r > 2. Indeed if Conjecture 16.61 is true, 
then the C r "closing lemma" for area preserving C r diffeomorphisms of S 2 holds 
when r > 16, because any any C r diffeomorphism could be C r approximated 
by a Moser generic one with dense periodic points. 

The strongest current result in the direction of Coniecture l6.6l the following. 

Theorem 6.7 (Franks and LeCalvez [10j). Suppose f : S 2 — > S 2 is an area 
preserving diffeomorphism which is Moser generic. Then there are countably 
many instability sets and their union is dense in S 2 . Equivalently, if Hyp(f) 
denotes the set of hyperbolic periodic points, then both 



are dense in S 2 . 

As a result of this, in order to prove Coniecture l6.6l it would suffice to show 
that the periodic points are dense in any instability set, or equivalently that the 
homoclinic points (i.e. points in W s {p) (~l W u (p)) are dense in the instability set 
associated to a hyperbolic periodic point p. At present this seems to be difficult 
however. 

For example, we know of no diffeomorphism with a hyperbolic periodic point 
p and an open set U in which both W s {p) and W u (p) are dense but with no 
periodic points and no homoclinic points in U. Remarkably, however, we don't 
know how to exclude this possibility even for Moser generic diffeomorphisms! It 
is difficult to imagine the structure of any such example, however. 

7 Properties of Instability Sets 

From its definition the instability set associated to a hyperbolic periodic point p 
of a Moser generic diffeomorphism / : S 2 — > S 2 is a compact set invariant under 
f n where n is the period of p. It has many other striking properties which we 
now summarize. 

Theorem 7.1 (Franks and LeCalvez |10|b Suppose f : S 2 — > S 2 is an 

area preserving diffeomorphism which is Moser generic and A is an f -invariant 
instability set. Then 

• All components of S 2 \ A are open disks. For each such disk U there is an 
n > depending on U such that f n {U) = U. 



U W s (p) and 



U wu (p) 



pGHyp(f) 



pEHyp(f) 
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• All periodic points in A are hyperbolic saddles. 

• A is the closure ofW s {q) for any q S Per(f) DA. Also A is £/ie closure of 
W u (q) for any q £ Per(f) A. 

Corollary 7.2. J/ A p and A q are instability sets associated with periodic points 
p and q respectively, then either A p = A q or A p n A q = 0. 

We can also consider the rotation numbers of points in an instability set. 
However, rotation numbers are not defined for points on S 2 so we must remove 
two fixed points to produce an (open) annulus. 

Proposition 7.3. Suppose A is an instability set and U is a component of S 2 \A 
satisfying f n (U) = U. Then f n has a fixed point in U . 

This is because f n : U — > U is an area preserving homcomorphism of an 
open disk. The Poincare recurrence theorem asserts that almost all points of U 
are f n recurrent and then the Brouwer plane translation theorem implies that 
/" has a fixed point. 

Suppose / : S 2 — > S 2 is a Moser generic diffeomorphism and that A is an 
/-invariant instability set. If Uq and U\ are components of the complement of 
A which are invariant under /", then we may remove a fixed point of f n from 
each to obtain an open /"-invariant annulus in which A is essential. Translation 
numbers and rotation numbers can be defined for an open annulus in precisely 
the same way as Definition 13.31 and Definition 13.51 or alternatively, we could 
"blow up" the two fixed points rather than deleting them and obtain a closed 
annulus. 

We will denote by ph{f n , Uq, U\) the set of all rotation numbers of points in 
A measured in the annulus formed in this way. It is not difficult to see that all 
these rotation numbers do not depend on the choice of particular fixed points, 
but only on the disks Uq and U\ containing them. 

Theorem 7.4 (Pranks and LeCalvez [10 ). Suppose f : S 2 — > S 2 is a Moser 
generic diffeomorphism and that A is an f -invariant instability set. If Uq and 
U\ are any components of the complement of A which are invariant under f n , 
then Pa(/"j Uq, U±) is a non-trivial closed interval. Every rational number in 
Pa(/™, Uq,U\) is the rotation number (with respect to Uq and U\) of a periodic 
point in A. 

In other words, for any two components of the complement of A there is an 
entire interval of rates at which points rotate around them! 

8 Connections between frontier components 

An important class of area preserving annulus diffcomorphisms, with numerous 
applications the class of twist maps. 
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Definition 8.1. An area preserving diffeomorphism f : A — ► A is called a twist 
map if 




Figure 4: The lift F of a Twist Map 
Geometrically this means that a radial line segment 
Jo = {(x,t) E A | < t < 1} 
is twisted in a positive sense, or that in the lift a vertical line segment 

J = {(x,t) 6 A | < t < 1} 
is bent in the positive direction (see Figure 4). 

Theorem 8.2 (Birkhoff). If f : A — > A is a twist map, any essential compo- 
nent of the frontier of an invariant open set is the graph of a Lipschitz function, 
in particular it is a circle. Hence any essential component of the frontier of an 
instability set is topologically a Lipschitz circle. 

This is a very strong property which may be true more generally. However, 
it is unknown if for a twist map or a Moser generic diffeomorphism whether 
all components of the frontier must be topological circles. We do know that in 
the general (non-Moser generic, non- twist) case the frontier of an essential open 
invariant set can have components which are bad fractals. 

Theorem 8.3 (Handel There exists a C°° area preserving diffeomor- 

phism of the annulus with an essential invariant continuum which is a pseudo- 
circle. 

Despite the fact that we do not know if essential components of the frontier 
of an instability set are topologically circles, it was shown by Mather using 
Caratheodory's theory of prime ends that there is always a well defined rotation 
number for any component of the frontier of an instability set of a Moser generic 
diffeomorphism. Indeed Mather proves the following important result. 
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Theorem 8.4 (Mather [17 ). Any component of the frontier of a Moser 
generic diffeomorphisms contains no periodic points and has an irrational rota- 
tion number. 

Mather's result actually applies somewhat more generally than the Moser 
generic setting (see ^7] for details). 

For twist maps then, we have several interesting properties. If A is an insta- 
bility set which is essential in the annulus There are two distinguished compo- 
nents, X and Y, of the frontier of A, namely those which are also the frontier of 
the components of A\ A which contain the boundary components of A. They are 
the only components of the frontier of A which are essential in A. The beautiful 
result of Birkhoff, Theorem l8.2l above. asserts that these are embedded Lipschitz 
circles. Birkhoff also proved another striking result about these components. He 
showed that that if U and V are neighborhoods of X and Y respectively, then 
there are n,m > such that f n (U) n V ^ and f rn (V) DU ^ 0. This im- 
plies that one cannot dynamically separate X and Y in the sense that any open 
invariant set containing one of them must contain the other in its closure. In 
particular there can be no invariant circles separating them. 

This result was subsequently significantly strengthened by John Mather who 
proved the following. 

Theorem 8.5 (Mather [16] ) - If f '■ A — > A is a twist map and X and Y are 

the two essential components of the frontier of B , then 

W S (X) n W U {Y) ^ and W U {X) n W S (Y) + 0. 

It would be nice to know the analogous statement for other components of 
the frontier of an essential instability set. It would also be nice to know this with 
hypotheses more general than those for a twist map. For example, an instability 
set of an area preserving diffeomorphism of S 2 has no distinguished frontier 
components and no distinguished co-ordinates in which to measure twist. At 
least in the Moser generic case we have precisely this result. 

Theorem 8.6 (Franks and LeCalvez [10 ). Suppose f : S 2 — > S 2 is Moser 
generic, and A is an instability set. If X and Y are any two components of the 
frontier of A, then 

W S (X) n W U (Y) ^ and W U (X) n W S {Y) 0. 

In some ways the components of the frontier of an instability set are like 
periodic points (for example, they are periodic). Pursuing this analogy we 
should compare Theorem 18. 61 to the following result. 

Theorem 8.7 ( D. Pixton [IS], C. Robinson [19j). Suppose f : S 2 -> S 2 
is Moser generic, and A is an instability set. If p,q £ A are periodic points, 
then 

w s ( P )nw u (q) ^0. 
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9 Questions about Moser generic diffeomorphisms 



The study of Moser generic diffeomorphisms (or any other residual subset of 
C r area preserving diffeomorphisms with r > 2) is a area with many more 
questions than answers. The results above as well as some additional results in 
|1(J| provide only a start in this investigation. 

Here we list some open questions about instability sets. They are framed 
in the context of Moser generic diffeomorphisms, but answers would be very 
interesting for any large class of C r diffeomorphisms with r larger than 1. It 
may be useful view these questions after a further look at the computer image 
of Figure 3. 

For all these questions we suppose / : S 2 — > S 2 is a C r , r > 2 Moser generic 
diffeomorphism preserving Lebesgue measure and let A be an instability set. 
Instability sets share some properties of hyperbolic sets (though they are never 
hyperbolic except in the case of an Anosov diffeomorphism on the torus). In 
particular it may be that instability sets are topologically transitive and have 
dense periodic points. 

Question 9.1 (Transitivity). Is there always a point in A whose orbit is dense 
in A? Are the periodic points of A always dense in A? 

An affirmative answer to the second question combined with Theorem 16.71 
would, of course, imply Coniecturc l6.6l However, it is conceivable that this con- 
jecture could be true without the periodic points being dense in each instability 
set. 

Question 9.2 (The frontier components). Are the components of the fron- 
tier of A topological circles ? Is the restriction to such a component a minimal 
homeomorphism ? 

It is quite possible that even when a component of the frontier is a circle 
that the restriction of / is not minimal, despite the fact that it has irrational 
rotation number by the result of Mather cited above (Theorem l8.4() . 

Question 9.3 (The size of instability sets). Does every instability set A 
have positive measure? Can A have interior? Can the complement of A have 
finitely many components? Indeed, must the complement even be non-empty? 

Of course, for diffeomorphisms on the torus T 2 we have the example of linear 
Anosov diffeomorphisms. These are Moser generic and have a single instability 
set equal to all of T 2 . This seems likely to be a phenomenon which cannot occur 
on S 2 . 

Closely related is the following question which really displays the depth of 
our ignorance about Moser generic diffeomorphisms. 

Question 9.4 (Elliptic Points). // / : S 2 — > S 2 is a Moser generic C r , r > 
2 diffeomorphism preserving Lebesgue measure, must it have infinitely many 
elliptic periodic points? Indeed must it have at least one elliptic periodic point? 
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